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Abstract 

This  investigation  was  a  general  attempt  to  determine 
if  Logistics  Support  Analysis  (LSA)  process  effectiveness 
could  be  measured  within  the  confines  of  the  Aeronautical 
Systems  Division.  Surveys  of  opinion  were  sent  to  LSA 
program  managers  representing  many  programs  and  general 
attitudes  were  collected  in  response  to  forty-three 
measurement  questions.  Response  packages  were  returned  by 
twenty-five  of  these  individuals.  The  survey  findings  were 
sorted,  categorized,  and  analyzed  against  a  background  of 
five  major  research  questions  ranging  from  the  measurability 
of  LSA  process  effectiveness  to  the  predictive  factors 
contributing  to  increased  levels  of  overall  effectiveness. 

Results  from  the  study  clearly  indicate  a  preponderance 
of  evidence  suggesting  that  LSA  process  effectiveness  is 
indeed  measurable.  While  this  is  clearly  the  first 
significant  step  in  modeling  the  effectiveness  issue,  the 
research  goes  on  to  identify  accepted  predictive  factors  to 
be  used  in  program  assessment.  Several  categories  of 
factors  were  examined  and  offer  a  point  of  departure  for  in- 
depth  variable  analysis.  Finally  the  study  investigates  the 
qualifications  of  individual  program  managers  and  suggests 


A  SURVEY  OF  EFFECTIVE  MEASURES  IN  THE 
LOGISTICS  SUPPORT  ANALYSIS  PROCESS 


I .  Introduction 

General  Issue 

Logistics  Support  Analysis  (LSA)  is  a  methodology  used 
within  the  bounds  of  the  system  engineering  process  to 
identify  and  analyze  the  support  resources  required  for  an 
evolving  weapon  system.  As  such,  LSA  can  be  used  as  a 
systematic  analytical  process  to  determine  both  quantitative 
and  qualitative  support  requirements  associated  with  the 
operations  and  maintenance  of  a  particular  piece  of 
equipment.  The  process  begins  early  in  the  concept 
exploration  phase  and  extends  throughout  the  production  and 
deployment  of  the  weapon  system.  Specifically,  the  LSA 
process  is  characterized  by  a  number  of  detailed  logistics 
tasks  that  form  the  foundation  for  very  complex  analytical 
trade  studies  and  logistic  support  assessments.  Since  the 
LSA  process  is  lengthy  and  can  easily  be  misapplied,  it 
requires  dedicated  and  technically  competent  supervision. 

Background 

In  recent  years  managers  within  the  Department  of 
Defense  (DOD)  have  met  with  both  exciting  and  challenging 
times,  especially  in  the  area  of  weapons  system  acquisition. 
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Past  government  policy  allowed  the  engineering  community  at 
large  to  "establish  the  program  direction"  for  the 
acquisition  of  major  military  and  government  hardware. 
Unfortunately,  this  kind  of  situation  has  resulted  in 
development  programs  that  reach  the  boundary  of  the  human 
technological  capacity,  with  little  recognition  or  regard 
for  the  practical  issues  of  weapons  system  support. 
Government  officials  now  realize  that  dwindling  resources 
and  an  increasing  national  deficit  prevent  the  U.S.  from 
expending  large  sums  of  money  toward  developing  systems 
without  adhering  to  a  minimal  life  cycle  cost  point  of  view. 
Managers  now  understand  the  importance  of  investing  in 
logistic  support  programs.  These  programs  are  important 
because  they  help  to  reduce  life  cycle  cost  by  improving 
reliability  and  maintainability,  and  by  increasing  overall 
system  availability.  Recent  research  has  shown  that 
operation  and  maintenance  costs  often  exceed  acquisition 
cost  during  the  first  f ive-to-seven  years  of  an  equipment’s 
life;  this  trend  continues  to  worsen  (5:1).  Many  older 
systems  in  the  U.S.  Air  Force  inventory,  like  the  F-4  and 
the  B-52,  reflect  the  same  trend.  As  one  individual  pointed 
out: 

Since  1 9 67 »  the  operating  and 
support  cost  for  each  hour  we  fly  an 
aircraft  have  quadrupled....  Over  a  20  year 
period,  it  has  been  estimated  that  the  Air 
Force  spends  two  dollars  on  operating  and 
support  costs  for  every  dollar  spent  to  buy 
the  weapon  system;  i.e.,  the  cost  of 
research,  development,  and  production  [5:1]. 
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DOD  agencies  have  now  taken  responsibility  to  reverse  the 
above  trends  by  issuing  policy  documents  for  use  in  the 
acquisition  of  military  weapons  systems.  Directive  5000.1 
is  first  in  order  of  precedence  for  management  of  systems 
acquisition  and  has  seniority  over  all  other  DOD  and 
military  regulation  (12:1).  The  document  establishes 
guidelines  for  the  management  of  major  system  acquisitions 
and  assigns  to  the  Defense  Systems  Acquisition  Review 
Council  the  responsibility  of  advising  the  Secretary  of 
Defense  on  decisions  of  importance. 

Another  directive,  DOD  5000. 3 9 »  establishes  policy  for 
the  development  and  management  of  an  Integrated  Logistics 
Support  (ILS)  program  (12:3).  This  document  explains  the 
need  for  evaluating  system  supportability  issues  and 
bringing  them  to  the  forefront  of  program  management 
responsibility  as  design  constraints  coequal  with 
performance,  cost,  and  schedule.  To  fully  understand  the 
significance  o*’  support  issues  and  their  impact  on  LSA 
process  effectiveness,  the  following  explanation  of  ILS  and 
LSA  are  provided. 

Integrated  Logistics  Support.  ILS  is  a  program 
management  concept  that  is  concerned  with  integrating 
logistics  into  the  design  of  equipment,  optimizing  the 
system  for  effectiveness,  and  minimizing  the  overall  life 
cycle  cost.  The  objective  of  an  ILS  program  is  to  field  a 
system  that  meets  the  predetermined  readiness  objectives  at 


an  affordable  cost.  The  reader  may  wonder  how  this  is 
accomplished.  Primarily  logistics  goals  may  be  attained  by 
designing  desirable  support  and  maintenance  characteristics 
into  the  system  or  equipment  being  developed  (10:2-2).  The 
above  task  may  be  as  simple  as  consulting  a  logistician  for 
general  advice  during  design,  or  as  complex  as  performing  a 
detailed  analysis  to  determine  the  optimal  equipment 
configuration  for  a  reliable  design  that  is  easily 
maintainable  in  the  field.  ILS  may  be  summed  up  by  saying 
that  it  represents  a  structured  iterative  approach  to  the 
management  of  various  technical  activities  which  are 
necessary  to: 

a.  Integrate  support  considerations  into  system 
and  equipment  design. 

b.  Develop  support  requirements  that  are  related 
consistently  to  readiness  objectives,  to  design, 
and  to  each  other. 

c.  Acquire  the  required  support. 

d.  Provide  the  required  support  during  the 
operational  phase  at  minimum  cost  [10:2-2]. 

Central  to  the  ILS  program,  the  Logistics  Support 
Analysis  (LSA)  process  is  a  key  technique  that  has 
evolved  to  accomplish  many  of  the  trade  off  analyses 
between  design  and  support  considerations  that  are 
needed  to  achieve  ILS  objectives. 

Logistics  Support  Analysis.  Because  of  the  extensive 
nature  of  the  process,  LSA  remains  an  often  misunderstood 
and  confusing  concept.  There  are  a  number  of  existing 
definitions  for  LSA;  however,  AFLCP/AFSCP  800-31*  clearly 


states  that  the  objective  of  LSA  is  to  structure  a  process 
which  systematically  pulls  together  all  engineering 
functions  that  contribute  to  the  design,  development,  and 
deployment  of  an  integrated  logistics  system  (11:10-1). 

The  LSA  process  begins  early  in  the  concept  exploration 
phase  of  the  acquisition  cycle  and  is  applied  throughout 
the  development  process  and  on  into  production  and 
deployment . 

The  LSA  process  represents  a  composite  of  systematic 
actions  taken  to  identify,  analyze,  and  influence  decisions 
made  about  logistics  support.  Initially,  the  thrust  of  LSA 
is  to  identify  quantitative  and  qualitative  objectives  for 
logistics  support.  The  LSA  objectives  later  become  the 
basis  for  design  parameters  which  are  translated  into 
hardware  development.  LSA  provides  the  input  needed  for 
performance  specification  and  the  design/cost  tradeoff 
analyses  that  are  conducted  on  an  iterative  basis.  As  an 
iterative  process,  various  LSA  analyses  (e.g., 
risk , logistic  support,  operational  availability)  are 
conducted  repeatedly. 

Early  in  program  development,  LSA  is  used  in  the 
evaluation  of  alternative  hardware  designs  and  their  impact 
on  support  postures  and  operational  readiness.  Design- 
related  risks  and  constraints  on  logistic  resources  like 
material,  personnel,  and  training  are  identified  and 
optimum  alternatives  are  selected  at  this  point.  As  the 


program  progresses  and  the  design  configuration  becomes 
fixed,  LSA  provides  timely  information  for  all  areas  of 
Integrated  Logistics  Support.  For  example,  analysis  of  the 
weapons  system's  configuration  can  provide  the  detailed 
information  needed  to  identify  not  only  the  support 
equipment  needed,  but  the  personnel  specialty  skills 
required  as  well.  This  information  is  needed  to  describe, 
and  put  into  place,  a  support  system  that  meets  the  LSA 
requirements  (6:5).  Finally,  LSA  data  is  stored  for 
historical  purposes;  this  comparative  tool  can  be  used  as  a 
design  baseline  to  influence  decisions  made  on  future 
systems.  Continuous  feedback  and  corrective  actions  make 
the  LSA  process  valuable  during  the  system's  life  cycle  and 
long  after  it  ceases  to  exist.  As  with  all  tools,  LSA  has 
the  potential  to  make  significant  contributions  in  the  form 
of  positive  enhancements  to  the  acquisition  of  a  weapons 
system,  but  this  applies  only  when  there  are  diligent 
people  who  understand  the  process  and  are  flexible  enough 
to  tailor  it  to  specific  program  needs. 


Specific  Problem 

In  recent  years,  the  Department  of  Defense  (DOD)  has 
required  all  major  acquisition  programs  to  establish  LSA 
programs  and  designate  LSA  program  managers.  This 
requirement,  combined  with  the  increasing  difficulty 
associated  with  effective  application  of  the  LSA  process 
has  caused  concern  within  the  acquisition  logistics 
community.  The  joint-command  guidance  office  for  LSA 
management  (AFALC/PTA)  has  expressed  serious  concern  over 
the  effectiveness  of  LSA  program  management  within  Air 
Force  Systems  Command  Product  Divisions.  Little 
information  has  been  collected  which  discusses  the 
competency  of  LSA  program  managers  or  the  effectiveness  of 
process  application.  Within  the  Aeronautical  Systems 
Division  (ASD),  there  is  no  substantial  data  base  from 
which  to  draw  conclusions  about  the  degree  to  which  the  LSA 
process  is  begin  applied  effectively  to  ASD  programs. 
Measures  of  effectiveness  need  to  be  identified  for  the 
purpose  of  measuring  the  LSA  program  management  process. 

Research  Questions 

1.  Can  LSA  process  effectiveness  be  measured? 

2.  What  level  of  effectiveness  can  be  assigned  to 

the  LSA  programs  surveyed  in  ASD? 


3.  Which  factors  can  be  identified  that  may  aid 
in  measuring  increased  levels  of  LSA  effectiveness 
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II.  Literature  Review 


Background 

Government  efforts  to  infuse  consideration  of  the 
complex  constraints  of  logistics  support  into  the 
engineering  design  process  have  been  underway  for  many 
years.  As  early  as  1964,  DOD  Directives  and  instructions 
established  a  requirement  for  each  service  to  implement 
some  type  of  Logistics  Support  Analysis  Program  (3:1)* 

This  early  attempt  to  formulate  government  policy  was  met 
with  separate  efforts  by  each  of  the  services  to  develop 
and  implement  the  DOD  Directive.  Unfortunately,  there  was 
no  collective  effort  to  accomplish  objectives  and  no 
unified  standard  specifying  how  LSA  was  to  be  achieved. 
Within  Air  Force  circles,  the  Ballistic  Systems  Division 
(BSD)  prepared  a  detailed  set  of  LSA  requirements  for  the 
Minuteman  Program  (3:1).  The  Navy  similarly  developed 
Maintenance  Engineering  Analysis  (LSA  equivalent)  and 
applied  the  process  to  both  the  F-4  and  F-111  Programs 
(3:1).  Air  Force  attempts  to  apply  the  LSA  processes 
initially  met  with  little  success.  In  summary,  the 
Logistics  Support  Analysis  process  during  the  1960's  was 
crude  and  only  marginally  effective  in  accomplishing  stated 
objectives . 

In  the  early  1970’s,  the  services  met  and  decided  to 
establish  a  single  source  of  guidance  for  the  application 
of  LSA.  The  resulting  document  was  a  military  standard  for 


Logistics  Support  Analysis  (MIL-STD-1388-1) ,  dated  15 
October  1973*  The  new  standardestablished  criteria 
governing  performance  of  Logistics  Support  Analysis  as  an 
integral  part  of  the  engineering  process  and  the  definition 
of  support  system  requirements  (6:1).  Although  not  fully 
successful,  the  standard  did  provide  for  an  integrated 
investigation  by  the  joint  services  into  the  structured 
requirements  of  LSA.  Because  of  obvious  inadequacies,  the 
first  military  standard  was  rewritten  to  result  in  the  new 
MIL-STD-1388-1 A.  MIL-STD-1 388-1 A  provides  a  single 
approach  for  all  military  services  to  conduct  and  document 
logistics  support  analysis  as  part  of  the  Integrated 
Logistics  Support  Program  (12:3).  The  revised  document  is 
intended  for  use  by  government  agencies  and  defense 
contractors  in  establishing  military  LSA  programs.  The  new 
emphasis  for  logistics  analysis  is  now  centered  around  the 
determination  of  manpower,  personnel  and  training 
requirements . 

Logistics  Support  Analysis 

Generally  the  Air  Force  Request  For  Proposal  (RFP)  is 
the  first  cpmmunication  with  industry  that  contains 
requirements  for  LSA  (11:10-1).  The  RFP  is  sent  to 
potential  contractors  requesting  a  proposal  describing  how 
each  contractor  would  accomplish  the  stated  government 
requirements.  Included  in  the  RFP  are  logistic  design 
parameters  like  Mean  Time  To  Repair  (MTTR)  and  Mean  Time 


Between  Maintenance  (MTBM)  that  communicate  system  goals  to 
defense  industry  participants.  The  responding  contractor 
firms  must  describe  (often  in  an  LSA  plan)  how  the 
contractor  proposes  to  apply  the  LSA  process  and  how  the 
systems  engineering  process  will  incorporate  the  functions 
of  the  LSA  program  (11:10-1).  The  contractor's  response 
must  be  descriptive  enough  to  communicate  all  the  details 
of  the  LSA  program  and  their  applicability  to  total  system 
objectives.  Subsequent  evaluation  for  contract  award  is 
based  partially  on  past  experience  but  primarily  on  how 
well  the  contractor  understands  and  intends  to  implement 
the  LSA  process. 

Placing  LSA  on  contract  represents  the  next  major 
milestone  in  the  series  of  process  requirements.  While  the 
negotiated  contract  marks  the  beginning  of  contractor 
responsibility,  it  does  not  mark  the  beginning  of  the 
analysis  process.  Some  form  of  support  analysis,  at  least 
at  a  topical  level,  is  usually  conducted  during  the  concept 
exploration  phase.  Upon  contract  award  during  full  scale 
development,  the  contractor  should  be  provided  with  any  and 
all  logistics  information  that  defines  the  expected  support 
posture . 

An  underlying  concept  of  LSA  is  that  the  contractor 
statement  of  work  is  different  for  each  contract. 
Flexibility  is  key  when  determining  which  LSA  tasks  are  to 
be  applied  to  meet  specific  needs  of  the  program.  For 


example,  the  LSA  task  requirements  for  commercial  "off-the- 
shelf"  equipment  would  be  much  different  from  the  tasks 
associated  with  development  of  new  design  equipment  (2:4). 
This  fundamental  difference  between  acquisition  strategies 
is  reflected  in  the  MIL-STD-1388-1A  tasks  that  are  applied 
to  each  LSA  contract. 

Despite  the  flexibility  involved  in  applying  LSA  tasks 
to  contracts,  some  activities  are  required  regardless  of 
the  type  of  acquisition.  The  Air  Force  Guide  For 
Supportability  Analysis  and  Supportability  Analysis  Record 
suggests  that  support  task  requirements  be  identified  by 
three  analysis  techniques:  (1)  detailed  review  of  system 
functional  support  requirements;  (2)  failure  modes, 
effects,  and  criticality  analysis  ( FMECA) ;  and  (3> 
reliability  centered  maintenance  (RCM)  analysis  (2:6). 
Failure  modes  and  effects  analysis  generally  tends  to  cause 
the  most  confusion.  In  the  performance  of  the  FMECA,  the 
contractor  must  identify  the  various  possible  modes  of 
failure  for  a  piece  of  equipment  and  then  determine  the 
effect  or  criticality  of  such  a  failure.  This  kind  of 
analysis  is  central  to  the  LSA  process.  Without  proper 
identification  of  possible  failure  modes,  there  is  little 
chance  of  documenting  and  analyzing  the  support  required  to 
maintain  an  operational  system.  Of  the  other  analysis 
techniques,  RCM  provides  a  systematic  means  of  determining 
the  feasibility/desirability  of  different  preventative 


maintenance  tasks,  thus  highlighting  maintenance  problem 
areas  for  design  review  consideration  (2:6). 

After  final  contract  award,  there  is  a  30-M5  day 
period  during  which  the  Air  Force  is  required  to  meet  with 
the  contractor  to  discuss  the  significance  of  the  LSA 
program  within  the  contractor's  systems  engineering  process 
(11:10-2).  The  guidance  conference  meeting  provides 
industry  with  an  opportunity  to  clarify  Air  Force  support 
concepts  and  fully  explain  their  approach  to  satisfying  LSA 
program  requirements.  Specifically,  the  guidance 
conference  provides  a  means  of  establishing  initial  LSA 
procedures  and  conditions  under  which  documented  results 
may  be  reviewed  and  validated  (2:7).  There  is  also  ample 
opportunity  to  discuss  how  the  LSA  data  will  be  passed  to 
those  authorized  government  agencies  who  request  summary 
reports.  Contractors  should  identify  both  organizational 
and  functional  management  responsibilities  for  LSA  and 
establish  the  level  of  indenture  (work  breakdown  structure) 
to  which  the  support  analysis  is  to  be  performed.  Lastly, 
both  parties  must  agree  to  the  equipment  items  that  will 
initially  be  identified  as  analysis  candidates.  From  this 
point  forward,  the  contractor  assumes  major  responsibility 
for  the  performance  of  LSA  tasks  and  the  generation  of 
acceptable  data  products. 

The  last  major  phase  of  activity  involves  conducting 
the  LSA  program  as  described  in  the  approved  contract.  In 


this  phase,  primary  activities  include  identification, 
quantification,  and  analysis  of  resources  needed  to  provide 
cost-effective  logistics  support.  In  reality,  the  LSA 
process  represents  a  two-component  process.  Initially, 
emphasis  is  placed  on  the  gathering,  analyzing,  and 
processing  of  information.  Later,  the  emphasis  shifts  to 
the  management  of  technical  efforts  to  produce  various  ILS 
products  and  services  (13:3).  The  result  is  that  the 
components  are  actually  indistinguishable  in  their 
contribution  to  the  integration  of  functional  logistics 
disciplines  (13:3)*  Graphically,  Figure  1  3hows  the 
application  of  LSA  is  characterized  by:  (1)  its  direct 
impact  on  total  logistics  support  for  a  given 
configuration,  and  (2)  the  feedback  effect  of ' the  logistics 
support  of  the  configuration  as  seen  below  (1:140). 


Figure  1.  Logistics  Support  Analysis  Application 


Often  overlooked,  the  single  most  important  component 
in  the  application  of  the  analysis  process  is  the  LSA 
analyst.  The  person  responsible  for  performing  the  LSA 
analysis  is  the  key  component  in  determining  what  support 
is  required  to  keep  the  system  operating  efficiently.  A 
very  real  problem  is  the  fact  that  there  are  not  enough 
qualified  individuals  to  adequately  perform  the  tasks  of 
the  LSA  process.  These  individuals  must  be  sufficiently 
familiar  with  engineering  to  understand  the  technical 
aspects  of  the  design/logistics  system  to  estimate  spares, 
support  equipment,  training,  manpower,  and  maintenance 
required  in  the  field  (8:1).  The  very  nature  of  LSA 
suggests  that  hardware  design  features  must  evolve  from  a 
detailed  list  of  specification  requirements  which  include 
logistics  constraints.  This  task  must  be  accomplished  on 
an  iterative  basis.  The  real  dilemma  arises  from  a  complex 
set  of  decisions  about  alternative  design  approaches  that 
must  be  offered  and  examined  throughout  system  acquisition. 
The  LSA  analyst  is  faced  with  the  question  of  when  should 
logistics  influence  on  design  cease  to  affect  the 
configuration  and  begin  to  establish  system  support 
requirements.  There  is  also  the  question  of  how  the 
existence  of  incomplete  data  affects  current  trade  studies 
that  ultimately  recommend  future  support  item  requirements. 


Often  such  questions  leave  the  program  analyst  agonizing 
over  the  accuracy  and  appropriateness  are  sufficient  enough 
for  early  recommendation.  As  Michael  McCarthy  of  Northrop 
Corporation  suggests: 

Perhaps  the  most  critical  task  of  the  LSA 
manager  is  to  assess  the  relative  criticality 
of  accuracy  versus  timeliness  for  a  specific 
analysis  or  trade  study  and  to  decide  when 
sufficient  data  is  or  is  not  available  to  reach 
an  analytical  conclusion  [8:2]. 


At  some  point  the  analyst  must  realize  that  no 
decision,  regardless  of  the  certainty  of  information, 
provides  an  opportunity  to  be  absolutely  correct.  The 
consensus  is  that  an  analyst  must  be  aware  of  the 
limitation  of  his  analysis  results  and  must  communicate 
this  information  along  with  the  recommended  support 
suggestions.  Despite  original  estimates,  the  quantitative 
description  of  the  required  support  is  likely  to  change 
many  times  over  the  life  cycle  of  a  system. 

In  summarizing  the  LSA  process,  it  is  sufficient  to 
say  that  it  is  more  than  just  an  analysis  tool;  LSA 
represents  a  methodology  that  provides  an  integrated 
approach  to  the  definition  and  quantification  of  a  system’s 
logistics  support  requirements.  With  the  cost  of  system 
support  doubling  the  cost  of  acquisition,  new  emphasis  must 
be  placed  on  recognizing  logistics  support  as  one,  if  not 
the  only,  prime  driver  in  future  contracting  and 
acquisition  policy.  Such  a  policy  change  is  a  must  if  the 


United  States  is  to  avoid  the  fate  of  rapidly  increasing 
system  costs— a  price  that  will  certainly  be  paid  by  the 
taxpayer.  For  many  people  the  choice  is  clear;  LSA  is  a 
requirement  on  all  procurements  and,  therefore,  must  be 
accepted  as  the  responsibility  of  program  management  for 
its  proper  application.  Individuals  outside  of  the 
logistics  community  must  be  held  accountable.  The 
implications  of  this  process  are  significant;  LSA  provides 
a  means  unlike  others  to  use  design  parameters  in 
determining  the  actual  numbers  of  required  spares,  skill 
requirements,  technical  data,  and  personnel.  In  short,  the 
synthesis  of  each  iteration  of  LSA  analyses  is  accomplished 
toward  the  eventual  achievement  of  one  singularly  important 
goal— cost  effective  support  for  design. 


III.  Methodolo 


Method  Justification 

The  analytical  techniques  characteristic  of  the  LSA 
process  involve  a  complex  series  of  trade-off  studies 
performed  by  contractor  personnel  to  determine  what  support 
is  juired  for  a  particular  system  or  equipment  item. 

Based  on  the  contractor  supplied  data,  the  government  LSA 
manager  is  required  to  screen  informational  inputs  in  an 
attempt  to  develop  a  coherent  logistics  strategy.  This 
strategy  involves  development,  procurement,  and 
distribution  of  a  "support  package"  properly  suited  to  meet 
the  life  cycle  needs  of  a  weapon  system  and  its 
corresponding  personnel.  As  such,  it  would  stand  to  reason 
that  one  of  the  individuals  most  familiar  with  the  ultimate 
objectives  and  success  of  the  LSA  process  is  the  person 
charged  with  the  responsibility  for  its  day-to-day 
application.  In  the  majority  of  cases,  the  responsible 
logistics  managers  are  assigned  to  the  offices  of  the 
Deputy  Program  Manager  for  Logistics  (DPML)  or  to  the 
Integrated  Logistics  Support  Management  (ILSM)  group  within 
the  appropriate  program  offices.  Within  the  acquisition 
community,  the  "front-line"  LSA  program  manager  is  often 
uniquely  qualified  to  make  a  valid  assessment  of  LSA 
process  or  program  effectiveness  because  of  continuous 
involvement  and  actual  management  responsibility  for  the 
process.  Along  with  the  organizations  responsible  for 


initial  operational  test  and  evaluation  and  those  who 
ultimately  maintain/support  the  weapon  system,  the  LSA 
manager  must  also  give  great  attention  to  assessing  the 
merits  of  a  support  program.  Evaluation  of  LSA  program 
effectiveness  in  often  based  upon  objectives  established  by 
the  original  logistics  support  manager.  Support  decisions 
made  throughout  the  years  will  be  based  upon  the  foundation 
laid  by  this  individual. 

Measurement  Instrument 

Characteristics  of  the  LSA  process  have  led  to 
selection  of  the  survey  instrument  as  a  means  of  collecting 
the  required  research  data.  Notwithstanding  various 
interview  techniques,  a  survey  represents  a  direct  approach 
to  collecting  the  reported  attitudes  of  a  selected  group  of 
LSA  managers.  Although  the  survey  technique  is  dependent 
upon  the  verbal  or  written  response,  its  versatility 
provides  an  avenue  for  collecting  information  on  opinion 
and  behavior.  This  fact  alone  provides  flexibility  for  the 
survey  to  be  used  as  a  valuable  measuring  instrument.  The 
survey  however,  is  not  without  its  shortcomings.  A  major 
disadvantage  of  a  survey  approach  is  the  lack  of  feedback 
the  respondent  receives  when  answering  questions.  In  the 
event  the  respondent  is  either  confused  by,  or  does  not 
understand  a  question,  there  is  no  means  by  which  to 
clarify  the  issue  with  feedback  from  the  person  issuing  the 
survey.  Additionally,  there  are  inherent  limitations  in 


using  rigid  sets  of  survey  questions  that  can  not  be 
altered  after  survey  issuance.  These  shortcomings  are  not 
necessarily  shared  by  other  techniques  (for  example,  the 
interview  method),  but  each  technique  does  have  its  own 
unique  set  of  disadvantages.  These  disadvantages  must  be 
carefully  weighed  in  selecting  and  developing  the 
appropriate  data  collection  strategy.  For  the  purposes  of 
this  study  the  survey  technique  was  selected  because  of 
limitations  on  the  amount  of  data  collection  time,  the  fact 
that  each  survey  is  exact  and  asks  the  very  same  questions 
of  every  participant  (unlike  the  interview  technique),  and 
because  surveys  appear  to  be  less  intimidating  than  verbal 
interviews . 


Sample/ Population 

Due  to  logistical  difficulties  involved  in  collecting 
research  data  from  remote  locations  and  the  limitations 
placed  upon  TDY  funding  for  student  travel,  this  study  has 
been  limited  to  the  ASD  community  at  Wright-Patterson  AFB. 
The  target  population  was  much  more  clearly  identifiable 
due  to  this  researcher’s  access  to  the  Wright-Patterson 
community  and  his  familiarity  with  ASD  organizational 
structure.  Potential  survey  candidates  were  identified  and 
selected  from  base  activities  responsible  for  logistic 
support  issues  within  the  ASD  program  offices.  According 
to  the  Air  Force  Acquisition  Logistics  Center  (AFALC) 
computer  records,  there  were  93  identifiable  DPML  or  ILSM 


offices  distributed  throughout  ASD.  Within  these  offices, 
the  LSA  manager  is  operationally  defined  as  the 
lndividual( s)  responsible  for  management  and  implementation 
of  LSA  activities  throughout  the  acquisition  cycle  of  a  DOD 
system.  Sampling  for  this  research  was  based  upon  a  rather 
straightforward  approach.  Since  cost  was  not  a  significant 
factor  in  the  selection  of  larger  and  larger  sample  sizes, 
selection  strategy  was  based  primarily  upon  selecting  as 
many  study  participants  as  could  be  identified  from  the 
existing  data  base.  This  sampling  technique  was  used  to 
identify  34  candidates  from  the  various  offices  responsible 
for  logistics  support  analysis  on  ASD  programs.  The 
selection  technique  also  has  the  added  benefit  of  achieving 
a  representative  sample  of  the  population  which  exemplifies 
many  of  the  possible  phases  in  the  acquisition  cycle.  This 
selection  of  programs  ranging  from  early  development  to  the 
latter  stages  of  production  reflect  a  direct  sampling  plan 
that  helps  to  minimize  the  influence  of  one  particular 
phase  on  the  final  conclusions  that  result  from  the  data. 
While  other  selection  techniques  like  straight-line  or 
weighted  methods  might  suffice,  the  objectives  of  this 
research  are  as  readily  served  by  the  above  procedure. 

Data  Collection 

The  survey  instrument  provided  in  the  appendix  to  this 
document  is  primarily  based  upon  an  existing  questionnaire 
obtained  from  the  AFALC  Guide  for  Supportability  Analysis 


and  Supportability  Analysis  Record.  From  this  guide, 
questions  concerning  the  contractual  applications  of  LSA, 
previous  training  and  experience  in  LSA,  and 
recommendations  for  improvements  were  used  to  provide  the 
survey's  foundation.  Additional  questions  about 
rank/grade,  official  position  title,  and  status  as  an  LSA 
point  of  contact  provided  the  much  needed  demographic 
information.  The  general  structure  of  the  survey  was 
arrived  at  based  upon  a  compromise  between  the  researcher's 
desire  to  answer  certain  basic  questions,  and  the  expert 
opinion  of  AFALC  staff  members.  Conscious  of  advisor 
input,  the  decision  to  include  or  exclude  questions  along 
with  the  effort  to  determine  their  sequence  resulted  from 
the  same  compromise  process.  As  for  survey  administration, 
the  survey  packages  are  to  be  sent  to  each  of  the 
candidates  identified  for  the  sample  group.  The  very 
structure  of  the  survey  requires  each  survey  participant 
respond  with  his  or  her  recommendations  for  adequate 
measures  of  LSA  process  and  program  effectiveness.  Because 
of  the  flexibility  for  each  participant  to  respond  as  he  or 
she  pleases,  there  is  a  possibility  for  dispersion  in  the 
responses.  To  prevent  problems  in  this  area,  the 
questionnaires  will  each  be  screened  and  the  answers  placed 
in  categories  to  be  analyzed  in  groups.  Group  categories 
will  be  established  upon  receipt  of  the  survey  data.  From 
this  categorization,  it  is  possible  that  the  observed 


IV.  Findings  and  Conclusions 


Introduction 

This  section  represents  a  summary  of  the  collective 
results  of  the  thesis  research  process.  The  text  is  divided 
into  five  sections,  each  corresponding  to  a  particular 
research  question  identified  in  the  introduction  (Chapter 
One).  The  proposed  research  question  will  be  presented  in 
each  section  along  with  actual  survey  results  in  categorized 
form.  Of  the  original  34  survey  packages  that  were  sent  out 
to  participating  ASD  offices,  a  total  of  25  completed 
surveys  were  returned  for  a  73-5%  response  rate.  The  raw 
data  from  all  surveys  is  contained  in  Appendix  B.  Of  the  25 
survey  forms  returned,  there  were  several  which  had  one  or 
more  measurement  questions  that  were  either  incomplete  (the 
individual  failed  to  respond  with  the  correct  number  of 
answers)  or  entirely  disregarded.  For  this  reason,  it  is 
apparent  the  summary  information  may  contain  data  which 
represents  a  total  of  less  than  25  survey  responses  per 
category.  In  this  case,  all  deductions  and  the  accompanying 
conclusions  are  based  upon  some  percentage  of  the 
actual  number  of  responses,  not  on  the  number  of  possible 


responses . 


Research  Question  One 


CAN  LSA  PROCESS  EFFECTIVENESS  BE  MEASURED? 

The  initial  basis  for  this  research  centered  around  a 
genuine  attempt  to  determine  whether  LSA  process 
effectiveness  could  accurately  be  measured.  Regardless  of 
subsequent  research  findings,  the  primary  concern  was  to 
survey  LSA  management  opinion  and  verify  the  measurability 
of  process  effectiveness.  Without  data  to  support  the 
measurability  assumption,  little  incentive  would  exist  to 
press  forward  with  additional  research  concerning  the  inputs 
to  and  results  of  effectively  structured  and  managed  LSA 
programs . 

Targeting  the  measuring  instrument  at  only  those 

individuals  responsible  for  managing  the  LSA  process,  survey 

question  28  was  particularly  structured  to  collect  data 

corresponding  to  the  above  research  question. 

After  close  scrutiny,  do  you  believe  the  effectiveness 
of  your  LSA  program  could  be  measured  in  either  quan¬ 
titative  or  qualitative  terms? 

a .  no 

b .  yes 

Statistical  inference,  the  process  of  making  judgment  about 
a  population  based  upon  examination  of  a  sample  was  the 
basis  used  for  modeling  the  measurement  question  for 
analysis.  The  decision  was  made  to  model  question  28  after 
a  binomial  distribution,  and  structure  the  model  upon  three 
major  assumptions: 


1.  Each  survey  is  independent  of  each  other. 

(Collusion  in  the  survey  responses  would  not  be  an 
expected  pattern  for  this  experiment.) 

2.  The  outcome  for  this  question  can  be  limited  to  one 
of  two  responses  (yes  or  no). 

3.  The  probability  of  a  yes  or  no  response  is  constant 
for  each  individual  survey. 

Each  of  these  assumptions  were  necessary  to  properly  conduct 
a  binomial  experiment. 

For  the  purposes  of  this  research,  a  hypothesis  testing 
approach  was  deemed  appropriate.  As  part  of  the  hypothesis 
testing  approach,  the  researcher  is  required  to  establish  a 
decision  criterion  for  the  population  and  its  constraint,  by 
which  subsequent  data  will  be  measured.  The  basic  decision 
criterion  established  that  the  majority  (greater  than  50%) 
of  the  survey  sample  would  have  to  respond  with  a  "yes" 
response  in  order  to  conclude  that  LSA  process  effectiveness 
can  be  measured  in  qualitative  or  quantitative  terms.  In 
addition,  a  null  and  alternative  hypothesis  must  be 
determined  for  the  experiment.  The  null  (denoted  by  Hq)  is 
the  hypothesis  assumed  to  be  true  until  there  is  sufficient 
evidence  to  the  contrary.  The  alternative  (denoted  by  H  ) 

3 

is  the  hypothesis  the  researcher  would  prefer  to  conclude. 
Using  the  50%  constraint  of  the  decision  criterion,  the  null 
hypothesis  stated  that  the  probability  of  collecting 
information  to  support  the  premise  of  process  effectiveness 
measurability  was  less  than  or  equal  to  50%.  The 
alternative  suggests  the  probability  of  collecting  the  same 

information  is  greater  than  50%. 
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(H0)  p<.50  (Ha)  p>.  50 

In  the  binomial  distribution,  X  is  defined  as  the 
random  variable  number  of  successes  or  "yes”  responses  to 
the  questionraire .  To  interpret  the  data  collected,  the 
researcher  must  first  determine  a  level  of  significance  for 
the  particular  analysis  test.  By  selecting  a  level  of 
significance  (  =  .15),  the  researcher  defines  the 

confidence  level  he  intends  to  be  satisfied  with.  In  this 
case,  selecting  an  o(  of  .15  corresponds  to  a  confidence 
level  defined  as  (1-.15)  which  means  that  those  conducting 
the  research  can  be  eighty-five  percent  sure  that  the 
conclusions  based  on  sample  observation  are  correct.  With 
all  assumptions/constraints  defined,  the  hypothesis  test  may 
now  be  conducted. 
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Statistical  Test 


The  appropriate  test  is  to  reject  the  null  hypothesis 

for  the  defined  significance  level  value  if: 

The  probability  of  the  random  variable  X  being  greater 
than  or  equal  to  the  observed  number  of  "yes"  responses 
(given  a  sample  of  25  and  a  probability  of  .50)  is  less 
than  the  significance  level  value 

EXAMPLE:  P  X>  16  |  25  ,  .5o]  <.15 

This  is  equivalent  to... 

1-p(X  <15) 

Using  the  A. 2  table  of  Cumulative  Probabilities  from  the 
Meek  and  Turner  text,  Statistical  Analysis  For  Business 
Decisions ,  the  above  equation  equates  to... 

1-. 8852=  .1148 

Comparing  the  computed  value  of  .1148  to  the  significance 
level  of  .10,  we  find  that  .1148  <C.  15,  and,  therefore,  we 
are  required  to  reject  Ho  (the  null  hypothesis). 

Conclusion 

The  results  of  the  hypothesis  testing  suggest  there  was 
sufficient  evidence  in  the  sample  to  reject  the  null 
hypothesis  and  conclude  that  the  probability  of  supporting 
the  premise  of  process  effectiveness  measurability  was 
greater  than  50%.  This  actually  means  that  the  majority  of 
LSA  managers  in  ASD  do  believe  that  LSA  process 
effectiveness  can  be  measured  in  qualitative  or  quantitative 
terms.  In  regard  to  the  research  at  hand,  this  is  a 
significant  statement  because  much  of  the  subsequent 
analysis  will  be  based  upon  the  fact  that  LSA  process 
effectiveness  can  indeed  be  measured. 


If,  however,  we  choose  not  to  believe  the  results  of 
the  statistical  test,  we  must  conclude  the  sample  used  was 
so  extreme  that  it  does  not  represent  the  population  at 
large.  A  new  sample  would  have  to  be  collected  and  the  test 
repeated.  With  a  limited  ability  to  radically  restructure 
the  sample  group,  and  no  evidence  of  a  uniquely  different 
sample,  the  conclusion  remains  unchanged.  The  majority  of 
LSA  program  managers  in  ASD  believe  in  the  measurability  of 
LSA  process  effectiveness. 

Research  Question  Two 

WHAT  LEVEL  OF  EFFECTIVENESS  CAN  BE  ASSIGNED  TO  THE  LSA 
PROGRAMS  SURVEYED  IN  ASD? 

Subsequent  to  the  question  of  whether  LSA  process 
effectiveness  can  be  adequately  measured,  a  secondary  issue 
exists  pertaining  to  what  levels  of  effectiveness  should  be 
associated  with  various  LSA  programs  within  ASD.  If  it  can 
be  proven  that  the  LSA  process  can  be  measured  and  that 
there  is  an  uncommonly  high  percentage  of  successful  (high 
level  of  effectiveness)  programs  within  ASD,  further 
research  might  provide  the  specific  predictors  needed  to 
model  the  effectiveness  process.  There  is  great  value  in 
having  the  capability  to  formulate  the  effectiveness  issue 
into  a  logistics  equation  or  model  into  which  predetermined 
factors  are  placed  to  predict  an  overall  level  of 
effectiveness  for  an  emerging  LSA  program. 
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Research  to  explore  this  idea  centered  around 
systematically  collecting  data  on  program  levels  of 
effectiveness.  The  objective  was  to  determine  what  level  of 
effectiveness  each  program  manager  associated  with  his 
particular  program.  Each  survey  participant  was  required  to 
answer  question  36  by  rating  the  overall  level  of 
effectiveness  on  their  particular  program,  based  upon 
individual  criteria  used  to  structure  the  original  program. 
Surprisingly  enough,  the  range  of  responses  to  question  36 
was  quite  wide  among  the  four  possible  responses  provided. 
Results  of  the  survey  are  summarized  below. 

SURVEY  QUESTION:  Based  on  the  four  criteria  you  cited 
in  question  29,  how  would  you  rate  the  level  of 
effectiveness  for  your  particular  LSA  program(s)? 


TABLE  I 

Summary  of  Program  Effectiveness  Ratings 


QUESTION  RESPONSES  NUMBER  % 

a.  exceeding  expected  performance  1  04 

b.  meeting  expected  performance  14  56 

c.  falling  behind  expected  performance  5  20 

d.  far  behind  expected  performance  1  04 

e.  DID  NOT  RESPOND  TO  THE  QUESTION  4 
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Summarized  in  the  data  above,  84%  of  the  people  who 
returned  surveys  responded  to  the  question  on  program  level 
of  effectiveness.  With  four  participants  choosing  not  to 
respond  to  the  question,  72%  of  those  who  responded  reported 
their  programs  were  currently  exceeding  or  meeting 
predetermined  levels  of  performance.  This  statistic  is  much 
higher  than  would  be  expected,  especially  in  light  of 
widespread  criticism  that  suggested  the  majority  of  LSA 
programs  currently  managed  within  ASD  are  in  fact  not 
meeting  predetermined  levels  of  performance.  Of  the 
remaining  responses,  24%  suggested  that  the  programs  were 
behind  expected  performance.  This  number  was  surprisingly 
low  given  the  original  assumptions  stated  above.  Lastly, 
those  choosing  not  to  respond  represented  only  16%  of  the 
returned  surveys. 

Conclusion 

To  avoid  control  of  the  research  results  by  program 
office  management  staffs,  the  surveys  were  sent  directly  to 
the  study  participants  and  returned  anonymously  by  mail. 
Because  of  this,  it  is  safe  to  assume  the  survey  responses 
were  not  influenced  by  the  coercion  of  management  staff 
members  but  represent  the  actual  opinions  of  each 
participant.  The  fact  that  only  24%  of  those  surveyed 
reported  their  programs  at  a  level  below  expected 
performance  suggests  that  the  vast  majority  of  LSA  programs 
are  measuring  up  to  expected  levels  of  performance.  This, 


combined  with  the  fact  that  program  managers  generally 
believe  in  the  measurability  of  process  effectiveness,  tends 
to  make  plausible  the  concept  of  a  set  of  commonly  shared 
effectiveness  predictors  among  successful  ASD  programs. 

Thus,  if  we  agree  LSA  process  effectiveness  can  be  measured, 
and  if  the  majority  of  programs  surveyed  are  above  or  at 
expected  levels  of  performance,  it  is  reasonable  to  conclude 
these  programs  may  share  a  common  group  of  factors  from 
which  process  effectiveness  can  accurately  be  measured.  If 
this  is  correct,  it  is  only  a  matter  of  further  research 
before  these  commonly  shared  predictors  are  identified, 
analyzed,  and  quantified  into  unique  measurement  elements. 
The  natural  extension  here  is  to  identify  those  factors  most 
commonly  reported  as  being  attributable  to  program 
effectiveness  and  to  analyze  them  for  underlying  trends. 

This  is  accomplished  in  the  following  section,  research 
question  three. 

Research  Question  Three 

WHICH  FACTORS  CAN  BE  IDENTIFIED  THAT  MAY  AID  IN  MEASURING 
INCREASED  LEVELS  OF  LSA  EFFECTIVENESS? 

Unlike  the  other  four  research  questions,  this  question 
was  designed  entirely  to  provide  qualitative  research  data 
for  this  study.  The  simplest  way  to  collect  data  on 
predictors  of  LSA  effectiveness  is  to  ask  for  a  listing  of 
those  factors  of  merit.  The  measurement  question  used  to 
provide  this  data  (survey  question  29)  deviated  from  the 
norm  by  requiring  each  participant  to  respond  by  listing  as 


many  as  four  factors  which  contributed  to  overall 
measurement  of  LSA  process  effectiveness.  The  purpose  of 
this  technique  was  to  determine  if  program  managers 
consistently  reported  many  of  the  same  effectiveness 
factors.  Answers  to  this  question  obviously  lead  us  closer 
to  solving  the  problem  of  analyzing  and  measuring  the 
relative  success  of  the  LSA  process.  For  example,  by 
correctly  identifying  the  factors  (predictors)  that 
contribute  to  measurement  of  effectiveness,  we  could  resolve 
many  of  the  difficulties  involved  in  structuring  a  model  to 
predict  LSA  process  effectiveness.  With  such  an  analytical 
model,  one  might  quantify  the  input  predictors,  run  a 
simulated  analysis  of  the  variables  and  constraints,  and 
offer  projections  of  program  success  at  given  intervals. 
There  is  certainly  great  value  in  having  this  capability. 

Originally,  the  intent  of  this  research  had  been  an 
attempt  to  define  the  parameters  necessary  to  construct 
actual  predictors,  for  input  into  an  LSA  effectiveness 
model.  However,  because  of  the  time  constraints  involved 
and  the  enormity  of  the  task,  there  was  insufficient  time  to 
perform  a  detailed  analysis  of  each  of  the  survey-reported 
factors.  Therefore  the  primary  task  was  to  collect  and  list 
each  of  the  response  types  provided  by  the  survey 
participants.  Since  eighty  percent  of  those  involved  in  the 
study  responded,  it  was  safe  to  assume  there  would  be  a  wide 
range  of  valid  responses.  Each  response  was  analyzed  for 


content  and  the  factors  separated  into  one  of  seven  general 
categories:  Analysis ,  LSA  Data,  Organizational  Dynamics , 

Scheduling ,  Process  Mechanics,  Contractor  Support,  and 
General  Support.  Categorizing  the  inputs  made  it  possible 
to  better  understand  the  relationships  among  factors.  In 
fact,  within  various  categories,  there  were  several  factors 
which  received  multiple  responses,  giving  further  credence 
to  the  concept  of  generally  accepted  measures  of  merit. 

These  results  are  contained  in  Table  2  at  the  end  of  this 
section . 

Each  category  was  labeled  based  on  the  general 
characteristic  of  the  factors  separated  into  that  particular 
group.  For  example,  if  the  factors  of  one  group  all 
suggested  the  importance  of  various  government  analyses 
techniques,  that  category  was  labeled  Analysis.  If  the 
factors  specifically  dealt  with  enhancing  the  LSA  process 
itself,  that  category  was  labeled  Process  Mechanics.  There 
were  three  categories  (LSA  Process  Mechanics,  LSA  Data,  and 
Analysis)  that  received  by  far  the  greatest  concentration  of 
responses.  These  three  categories  were  considered  most 
significant  because  they  were  cited  most  often  in  the 
surveys,  they  were  more  clearly  articulated,  and  they  could 
be  most  easily  related  to  the  LSA  Analysis  Process.  Clearly 
these  factors  would  be  first  choice  for  further  study  based 


on  the  above  criteria. 


In  the  Process  Mechanics  category,  the  responses  all 
contributed  to  enhancements  of  the  LSA  process  itself.  The 
responses  ranged  from  the  impact  design  factors  have  on  LSA 
to  the  need  for  developing  a  requirements  baseline.  The  LSA 
Data  category  represented  various  data  output  products  which 
result  from  the  analysis  process.  The  participants 
generally  suggested  that  improvements  were  needed  in  the 
areas  of  data  validity,  accuracy,  and  accessibility  in  order 
to  have  a  positive  impact  on  process  effectiveness.  The 
Analysis  category,  however,  was  quite  different  from  the 
other  two  categories.  Program  managers  cited  a  wide  range 
of  support  analysis  techniques  they  considered  crucial  in 
determining  LSA  process  effectiveness.  Survey  responses 
identified  deficiencies  in  analytical  techniques  ranging 
from  Failure  Modes  and  Effects  Criticality  Analysis  ( FMECA) 
to  Reliability  Centered  Maintenance  ( RCM) .  Clearly,  these 
appear  to  be  factors  which  could  be  used  to  measure  final 
levels  of  LSA  effectiveness. 

Excluding  the  Scheduling  category,  the  other  three 
categories  seemed  less  significant  as  valid  predictors  based 
on  the  criteria  of  frequency,  clarity,  and  direct  relevance 
of  the  responses  provided.  In  the  category  of 
Organizational  Dynamics,  the  responses  related  to  areas 
within  the  organizational  structure  that  could  be  altered  to 
affect  the  LSA  process.  These  factors  involved  such 
deficiencies  as  lack  of  management  control  over  logistics 


support  and  ineffective  information  flow.  Factors 
identified  under  the  General  Support  category  were  the 
miscellaneous  items  suggested  as  general  improvements  to  the 
Logistics  Supports  Analysis  process.  For  example,  a 
properly  delivered  support  system  and  the  timeliness  of 
support  actions  both  contribute  to  an  effective  weapons 
system  support  posture;  this,  of  course,  is  the  desired 
result  of  the  process.  But  it  is  almost  impossible  to 
measure  while  the  process  is  underway.  Lastly,  the  category 
of  Contractor  Support  typically  represented  those  factors 
under  defense  contractor  control  that  generally  contribute 
to  the  success  or  failure  of  the  LSA  process.  Contractor 
translation  of  LSA  parameters  and  the  internal  organization 
structure  are  factors  often  outside  the  scope  of  government 
influence.  These  factors  have  a  tremendous  impact  of  the 
eventual  outcome  of  the  program.  Further  research  will  have 
to  be  performed  to  determine  which  process  effectiveness 
factors  are  within  the  confines  of  defense  contractor 
control,  and  which  factors  significantly  impact  the  success 
of  the  entire  LSA  process. 

Conclusion 

Of  all  categories,  Scheduling  is  possibly  the  easiest 
source  of  measurement  predictors  to  quantify.  It  is  not 
difficult  to  link  such  things  as  milestone  dates  and 
functional  work-around  plans  to  structured  management  goals. 
The  successful  completion  of  an  LSA  objective  on  or  before 


the  scheduled  milestone  date  can  be  tied  directly  to  proper 
application  of  the  LSA  process.  The  same  is  true  for  a 
properly  implemented  work-around  plan  for  management 
problems.  Effective  analysis  of  the  suggested  factors  could 
result  in  a  quantitative  evaluation  technique  to  be  applied 
in  the  achievement  of  LSA  management  objectives.  The  level 
of  process  effectiveness  could  be  based  upon  a  rating  scheme 
that  applies  a  negative  score  to  both  late  and  extremely 
early  completion  of  various  sub-tasks.  Related  sub-tasks 
would  be  grouped  and  related  to  the  achievement  of  overall 
objectives.  The  result  of  such  research  would  be  a 
mathematical  model  to  be  used  in  the  actual  measurement  of 
LSA  process  effectiveness  as  related  to  the  ASD  program 
structure . 

In  the  Analysis  category,  fifty-five  percent  of  the 
identified  factors  represent  redundant  responses.  The 
effectiveness  of  any  LSA  program  is  significantly  impacted 
by  the  different  analyses  used  to  provide  the  input  data. 

As  such,  determining  a  measurable  link  between  the 
completeness  of  an  analysis  technique  and  actual  program 
results  is  the  kind  of  relationship  needed  to  identify  the 
quantitative  measurement  factors  in  this  area. 

More  significant  is  the  fact  that  seventy-five  percent 
of  all  factors  under  the  LSA  Data  category  are  multiple 
responses.  Overwhelmingly,  the  LSA  managers  suggest  that 
data  and  related  data  products  are  the  primary  concern 


regarding  measurement  of  process  effectiveness.  The  need 
for  consistency  among  the  various  LSA  output  products  was  a 
recurring  theme  in  many  of  the  surveys.  This  suggests  a 
need  for  a  single  standardized  logistics  data  base  of 
information.  There  are  obvious  benefits  in  this  single 
source  of  data.  A  measurement  instrument  must  be  developed 
to  include  those  data  characteristics  which  can  be  directly 
related  to  the  quality  of  information  provided.  By  doing 
so,  the  researcher  produces  an  overall  measure  for  LSA  Data 
that  directly  contributes  to  process  effectiveness. 


TABLE  II 


Factors  Contributing  to  LSA  Effectiveness 

SCHEDULING  (2  factors) 

Adhering  to  milestone  schedules 
Adopting  functional  work-arounds 

ANALYSIS  (11  factors) 

Maintenance  Task  Analysis  (2) 

Provisioning  Analysis  (2) 

Support  Equipment  Requirements  Analysis  (2) 

Failure  Modes  and  Effects  Criticality  Analysis 

Use  Study/Technical  Analysis 

Reliability  Centered  Maintenance  Analysis 

Network  Repair  Level  Analysis 

Early  Fielding  Analysis 

ILS  Resource  Analysis  (2) 

Task  Performance  Analysis  (2) 

Alternatives  Evaluation  Analysis  (2) 

ORGANIZATIONAL  DYNAMICS  (4  factors) 

Organizational  Interfaces 
Information  Flow 
Management  Attention  (2) 

Internal  Communications 


LSA  DATA  (8  factors) 

Accessibility  of  Data  (2) 

Accuracy  of  Data  (3) 

Consistency  Between  Documents  (2) 
Volume/Quantity  of  Data  (2) 

Data  Management  Discipline 
Validity  of  LSAR 

Availability  of  Review  Teams  (2) 
Currency/Quality  of  Data  (3) 

CONTRACTOR  SUPPORT  (3  factors) 
Parameter  Implementation 
Organizational  Structure 
DCAS  Interface 

PROCESS  MECHANICS-LSA  (10  factors) 

Influence  on  Design 

Requirements  Baseline 

LSA  Plan  Quality 

Coherent  Management  Program  (2) 

Maintenance  Concept  Definition 

Reliability/Provisioning  Factors  (2) 

Task/Skill  Analysis 

LSA  Program  Intent 

WBS  Involvement 

Design  Synthesis  Process 

i*0 
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Research  Question  Four 

WHAT  PERCENTAGE  OF  PARTICIPATING  LSA  PROGRAM  MANAGERS  SEEM 
QUALIFIED  TO  MAKE  JUDGMENT  ON  THE  ISSUE  OF  EFFECTIVENESS? 

The  aforementioned  research  questions  are  all  based 
upon  the  premise  that  those  people  responding  to  the  survey 
are  qualified  to  make  judgment  about  the  LSA  process.  At 
issue  here  is  the  fact  that  the  analysis  of  the  data,  and 
any  corresponding  conclusions,  are  the  result  of  inputs  and 
recommendations  offered  by  those  individuals  selected  for 
the  sample  group.  Should  a  significant  portion  not  be 
qualified  to  respond,  results  of  the  entire  research  might 
be  altered  or  skewed.  Possible  lack  of  respondent 
qualification  provides  an  unwanted  source  of  bias,  resulting 
in  the  possibility  of  unsubstantiated  conclusions  or 
recommendations.  Restricting  the  amount  of  bias  introduced 
into  the  study  provides  the  researcher  with  the  ability  to 
assess  the  results  and  offer  a  more  accurate  summary  of  the 
facts  as  they  exist.  To  establish  a  reference  point  for  the 
concept  of  the  qualified  survey  participant,  several 
assumptions  were  made  about  the  necessary  requirements.  The 
first  assumption  was  the  requirement  for  the  participant  to 
have  completed  at  least  one  formal  LSA  training  course.  To 
qualify,  the  individual  had  to  have  one  government  offered 
course  (i.e.  a  professional  continuing  education  course),  an 
LSA  course  offered  as  a  part  of  a  civilian  education 
program,  or  an  alternate  category  which  qualifies  as  formal 


LSA  training.  Secondly,  there  was  a  requirement  for 
previous  experience  in  the  role  of  LSA  program 
manager/support  analyst.  For  this  experiment,  an  arbitrary 
value  of  12  months  previous  experience  was  selected  as  the 
deciding  factor  for  qualification.  Lastly,  the  assumption 
was  made  that  it  would  be  difficult  to  properly  function  as 
an  effective  program  manager  without  being  adequately 
familiar  with  the  appropriate  DOD  and  Air  Force  regulations 
or  military  standards  concerning  the  procedures  for  proper 
application  of  the  LSA  process.  Familiarity  with  these 
documents  represents  the  final  criterion  for  qualification 
as  an  LSA  program  manager  capable  of  responding  to  this 
survey. 

The  following  survey  questions  were  selected  as 
corresponding  to  the  broad  issue  of  program  management 
qualification  mentioned  above: 

a.  Have  you  had  any  formal  training  and/or  education 
in  the  area  of  LSA,  the  LSA  Record,  or  the 
application  of  the  LSA  process?  (question  7) 

b.  How  effective  was  the  formal  education  or  training 
in  helping  you  to  improve  management  of  the  LSA 
process  on  your  particular  program(s)?  (question  8) 

c.  How  many  months  have  you  managed  or  worked  in  direct 
support  of  LSA  on  this  or  any  other  program? 
(question  5) 

d.  Are  you  familiar  with  the  primary  guidance  documents 
on  the  subject  of  LSA  program  management?  (question 
14) 

e.  How  would  you  describe  your  qualifications  to  make 
decisions  or  apply  judgment  to  rate  effectiveness 
of  LSA  within  your  program?  (question  10) 


These  questions  were  used  as  the  basis  for  development  of  a 
weighted  grading  scheme  which  could  be  used  to  quantify  the 
qualifications  of  each  individual  survey  participant  against 
an  objective  set  of  criteria.  After  detailed  review  of 
several  weighting  schemes,  the  following  factors  and 
weighted  values  were  assigned : 

1.  Completion  of  formal  training/education  (.10) 

2.  Reported  effectiveness  of  formal  training  (.10) 

3.  Reported  previous  LSA  experience  (.35) 

4.  Reported  familiarity  with  government  documents  (.10) 

5.  Self-reported  level  of  qualification  (.35) 

The  weighted  values  were  assigned  based  upon  a  determination 
that  previous  LSA  experience  and  self-reported  qualification 
were  the  two  most  important  factors  in  assessing  an 
individual's  overall  qualification.  The  other  three  factors 
were  less  important  than  the  first  two,  but  were  considered 
equally  important  among  themselves.  By  this  distribution,  a 
total  weight  of  1.00  (which  is  100  percent)  was  allocated  to 
the  appropriate  factors.  Each  survey  participant  was  rated 
against  the  above  factors  by  multiplying  the  score  (1  or  0) 
received  for  appropriate  survey  responses  by  the  weight 
factors.  In  this  scheme,  1  equates  to  a  "qualified" 
response,  while  0  equates  to  a  "not  qualified"  response. 

For  example,  on  survey  question  7  (formal  education)  the 
appropriate  response  for  a  qualified  individual  would  be  an 
a,  b,  or  d.  These  responses  represent  the  fact  that  the 
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respondent  has  had  some  form  of  formal  LSA  education 
(government,  civilian,  etc.).  The  only  non-qualifying 
answer  in  this  case  would  be  a  response  of  c,  "...  I  have 
never  had  any  formal  education."  The  appropriate 
"qualified"  and  "not  qualified"  responses  are  listed  for 
each  question  in  Table  III.  There  may  be  some  confusion  as 
to  how,  in  training  effectiveness  (survey  question  8),  a 
participant  who  did  not  respond  to  the  question  may  still 
receive  1  point  for  a  "qualified"  response.  You  must 
remember  that,  if  the  individual  replied  that  he  had  not 
received  any  formal  training  in  question  7,  he  certainly 
would  not  be  required  to  comment  on  the  effectiveness  of 
such  training  in  question  8.  In  this  case,  a  non-response 
was  classified  as  a  "qualified"  answer. 

The  next  step  involved  determining  a  cut-off  level  for 
the  total  computed  score  to  differentiate  the  "qualified" 
from  the  "not  qualified"  program  managers.  Those 
respondents  with  a  total  computed  score  of  .65  or  greater 
were  judged  as  being  qualified  LSA  program  managers.  The 
.65  cut-off  value  is  significant  in  that  a  respondent  must 
be  qualified  in  at  least  one  of  the  two  major  factors 
(previous  experience  or  self-reported  rating)  before  he  may 
receive  an  overall  rating  of  qualification.  In  addition,  if 
qualified  in  only  one  of  the  two  major  factors,  he  must  be 
qualified  in  all  of  the  remaining  three  areas  to  achieve  the 
minimum  .65  cut-off  value.  This  stringent  requirement  made 
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it  possible  to  select  only  those  participants  with  the 

highest  of  qualifications  for  a  •'qualified*'  rating.  The 

greater  the  percentage  of  "qualified"  survey  respondents, 

the  lower  the  amount  of  bias  introduced  into  the  study.  A 

composite  of  all  returned  surveys  is  provided  in  Table  III 

with  a  summary  of  the  responses  to  each  of  the  five  survey 

* 

questions . 

Conclusion 

After  compiling  all  of  the  data,  18  o i  25  survey 
respondents  ( 72% )  were  scored  as  "qualified"  LSA  program 
managers.  More  significant  was  the  fact  that  six  additional 
participants  could  have  easily  been  rated  as  fully  qualified 
if  the  cut-off  value  had  been  set  at  .55  rather  than  .65. 
This  ten  percent  difference  would  have  meant  the  respondent 
needed  to  be  qualified  in  one  of  the  major  factors  and 
two  of  the  three  additional  factors  in  order  to  achieve  an 
overall  "qualified"  rating.  This  is  significant  in  terms  of 
credibility  because  it  means  that  96%  of  the  survey 
respondents  fall  within  an  "essentially  qualified"  range. 

The  corresponding  conclusion  is  that  a  very  high  percentage, 
at  least  72%  but  possibly  more,  of  the  program  management 
opinions  found  in  this  study  represent  valid  data  input  and 
a  proper  source  of  information  for  assessing  the 
characteristics  of  the  LSA  process.  Assuming  this  is  a 
representative  sample,  the  opinions  confirmed  in  this  survey 
are  generally  characteristic  of  the  entire  ASD  population  of 
LSA  program  managers. 


Research  Question  Five 

HAVE  INDIVIDUAL  PROGRAM  REQUIREMENTS  BEEN  EFFECTIVELY 
TAILORED  AND  LEVIED  AGAINST  THE  APPROPRIATE  CONTRACTORS? 

The  fifth  and  final  research  question  revolves  around 
one  last  issue  of  concern.  Imagine  that  it  is  determined 
that  LSA  process  effectiveness  can  be  adequately  measured. 
However,  the  ASD  programs  being  measured  are  consistently 
rated  at  low  levels  of  effectiveness.  What  factor(s) 
contribute  to  low  program  effectiveness?  If  the  study  data 
shows  that  low  levels  of  overall  process  effectiveness  are 
neither  the  result  of  program  manager  qualifications  nor 
procedural  requirements,  it  may  be  proper  to  assume  that 
other  factors  are  responsible.  Past  evidence  suggests  that 
proper  application  of  LSA  program  requirements  might  be  a 
factor  of  importance.  To  test  the  question  of  general 
requirements  applications  and  the  related  issue  of 
requirements  application  on  sub-contractor  and  vendor 
contracts,  three  related  survey  questions  were  selected  to 
provide  measurement  information.  Each  survey  question  is 
constructed  with  a  possible  response  of  yes  or  no  and  is 
reprinted  below: 

a.  Have  the  requirements  for  LSA  been  appropriately 
tailored  to  meet  the  specific  needs  of  your 
program?  (question  16) 

b.  Is  there  a  need  for  additional  requirements 
tailoring  on  this  LSA  program?  (question  17) 

c.  Do  you  feel  the  requirements  of  your  LSA  program 
have  been  appropriately  levied  against  subcontractor 
or  vendors?  (question  18) 


Although  each  survey  participant  was  required  to  answer  each 
of  three  measurement  questions,  a  few  individuals  apparently 
chose  not  to  respond.  Results  of  the  participant  responses 
are  summarized  in  the  table  below. 

TABLE  IV 

Summary  of  Requirements  Application 


Survey  Question 

NO 

YES 

NO  RESPONSE 

16 

1 

23 

1 

17 

10 

13 

2 

18 

3 

21 

1 

Conclusion 

The  true  purpose  of  the  test  was  to  determine  if  there 
were  clearly  identifiable  trends  in  the  area  of  overall 
program  effectiveness.  Of  those  who  responded,  96%  reported 
that  the  original  LSA  requirements  levied  on  contract  were 
sufficient  to  meet  the  needs  of  their  particular  programs. 

As  such,  only  one  individual  chose  to  report  that  original 
requirements  were  inadequate.  This  fact  is  even  more 
significant  since  this  same  individual  still  rated  the 
overall  program  as  meeting  expected  levels  of  performance 
(question  36).  Overwhelming  evidence  suggests  that,  while 
original  LSA  requirements  are  being  correctly  applied  in  ASD 
programs,  there  appears  to  be  little,  if  any,  correlation 
with  levels  of  program  effectiveness.  As  such,  the  evidence 
suggests  no  causal  link. 
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with  levels  of  program  effectiveness.  As  such,  the  evidence 
suggests  no  causal  link. 

On  the  issue  of  additional  requirements  tailoring,  the 
responses  seemed  nearly  evenly  divided.  Forty-three  percent 
of  the  respondents  reported  there  was  no  need  for  additional 
tailoring  during  the  program's  life  cycle;  fifty-six  percent 
suggested  that  additional  tailoring  was  indeed  necessary. 
Given  a  general  pre-survey  assumption  that  low  program 
effectiveness  is  directly  attributable  to  a  lack  of 
requirements  tailoring,  one  might  assume  that  most 
individuals  reporting  a  need  for  additional  tailoring  would 
also  report  their  programs  as  being  below  expected 
performance.  This  was  not  the  case.  The  majority  of 
respondents  who  reported  the  need  for  additional 
requirements  also  suggested  that  their  programs  either  met 
or  exceeded  expected  levels  of  performance.  Once  again,  the 
expected  results  did  not  prove  to  be  true,  and  a  causal  link 
could  not  be  offered  as  a  contributing  factor  toward  low 
levels  of  program  effectiveness. 

The  last  issue  of  requirements  appropriately  levied 
against  sub-contractor/vendor  contracts  appears  to  be  more 
subtle.  Why  should  there  be  concern  for  requirements 
application  at  the  contractor  level?  Since  much  of  the  LSA 
equipment  and  trade-off  analysis  is  conducted  at  contractors 
sites,  the  potential  for  inadequate  requirements  application 
at  the  contractor  level  could  have  a  tremendous  impact  on 
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the  success  of  the  LSA  process.  Rather  than  avoiding  the 
issue,  this  research  was  purposely  constructed  to  try  to 
confirm  or  deny  the  existence  of  such  a  relationship.  Of 
those  who  responded,  88%  felt  program  requirements  had  been 
appropriately  levied  against  subcontractors  and  vendors;  13% 
voiced  an  opposite  opinion.  The  results  suggest  that  proper 
application  of  LSA  program  requirements  on  sub-contractors 
and  vendors  is  not  a  disrupting  factors  in  the  LSA  process. 
Moreover,  the  facts  actually  suggest  that  inadequate 
requirements  application  is  a  "non-issue".  Program  managers 
overwhelmingly  stated  that  LSA  program  requirements  had  been 
applied  correctly  on  their  particular  programs.  This  may  be 
the  result  of  a  general  unwillingness  of  the  managers  to 
admit  failure;  however,  there  is  no  evidence  to  support  this 
premise . 


V.  Recommendations 


The  preceding  research  effort  has  been  a  valuable 
learning  experience.  The  same  or  similar  survey  research 
should  be  conducted  within  the  U.S.  aerospace  defense 
contractor  community.  The  defense  contractor  is  a  vital 
link  in  the  support  analysis  process,  and  the  opinions  of 
their  management  organization  would  be  critical  in  any  plan 
for  improvement  of  the  process  as  it  now  exists.  The  same 
survey  instrument  should  be  used  to  allow  for  a  direct 
comparison  between  specific  participant  responses. 
Statistical  analysis  of  these  two  supposedly  disparate 
populations  could  later  identify  major  differences  (if  any) 
between  gov*  -nment  and  contractor  management  opinions  about 
the  LSA  process.  Regardless  of  the  survey  instrument  used, 
the  value  of  tapping  this  unused  resource  is  enormous. 

The  research  relating  to  research  question  four 
involved  determining  a  set  decision  criteria  to  establish 
qualifications  for  the  participating  LSA  program  managers. 

Of  those  individuals  rated  "not  qualified"  to  participate  in 
the  preceding  effort,  100  percent  were  found  deficient  in 
either  the  category  of  formal  training  or  previous 
experience.  Although  limited  in  scope  of  study,  a  need 
exists  for  a  formal  government  program  to  recruit,  educate, 
and  train  logistics/support  analysts  to  manage  LSA 
development  programs.  Efforts  should  emphasize 
identification  of  those  candidates  most  susceptible  to 
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logistics  training  (i.e.  managers  with  acquisition  or 
logistics  backgrounds).  The  Air  Force  should  develop  an 
education  program  that  is  capable  of  producing  efficient  and 
cost  effective  LSA  program  managers.  The  directorate  of 
Logistics  Support  Analysis  (Air  Force  Acquisition  Logistics 
Center)  is  central  to  this  issue  and  should  be  key  in  the 
development  process. 

This  research  determined  that  LSA  process  effectiveness 
can  indeed  be  measured.  Now  more  specific  research  should 
be  conducted  to  analyze  the  factors  identified  in  research 
question  three  to  determined  exact  and  quantifiable 
relationships  between  these  input  factors  and  measures  of 
LSA  process  effectiveness.  This  difficult  task  may 
ultimately  result  in  development  of  a  computer  model  for 
determining  process  effectiveness.  As  a  point  of  departure, 
it  is  recommended  that  the  researcher  begin  by  examining  the 
three  most  significant  groups  of  factors  because  of  their 
frequency,  clarity,  and  relevance.  An  effort  to  determine 
interrelationships  among  factors  would  be  most  productive  in 
these  areas.  However,  research  should  not  be  limited  to 
these  factors  alone,  analysis  of  other  identified  factors 
may  prove  to  be  profitable.  More  precise  opinion  questions 
should  be  constructed  to  measure  relative  priority  and  exact 
differences  among  factors;  this  is  necessary  for  more 
conclusive  results.  The  results  of  this  research  would 
represent  a  decisive  step  in  modeling  process  effectiveness, 
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while  understanding  the  determinants  of  overall  LSA  program 
success . 

Within  AFALC,  an  office  should  be  established  to 
initiate,  coordinate,  and  sponsor  logistics  research 
efforts,  specifically  for  Logistics  Support  Analysis. 
Emphasis  should  be  placed  on  injecting  the  responsibility 
for  such  activities  into  the  mainstream  of  Air  Force  Systems 
Command  research  and  development  projects  for  the  design  of 
future  weapon  systems. 

Notwithstanding  current  efforts,  new  enthusiasm  must  be 
generated  for  supporting  the  research  and  analysis  of 
controversial  logistics  issues  which  are  now  affecting  the 
Air  Force's  world-wide  support  posture.  Funds  should  be  set 
aside  to  sponsor  both  continuing  graduate  academic  research, 
and  to  provide  the  impetus  for  defense  contractor  study 
contracts.  While  it  may  be  more  cost  effective  to  sponsor 
in-house  study  efforts,  a  vast  resource  of  logistics 
expertise  resides  within  the  civilian  community  and  remains 
largely  untapped. 
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Appendix  A:  Survey  of  Opinion 

DEPARTMENT  OP  THE  AIR  FORCE 
HEADQUARTERS  AIR  FORCE  ACQUISITION  LOGISTICS  DIVISION  (AFLCI 
WRIGHT-PATTERSON  AIR  FORCE  BASE.  OHIO  45433 


nrPCY  TO 

ATTN  OWt  PTA 


•  UftJKCTs  Survey  of  Logistics  Managers 


to.  See  Distribution 


L  The  personnel  identified  on  the  enclosed  attachment  1  have  been  selected  to  participate 
in  an  ASD-wide  survey  for  logistics  related  information.  Each  individual  is  requested  to 
read  the  attached  survey  (Atch  2)  and  respond  to  each  of  the  questions  provided. 

2.  Response  to  this  AFALC  survey  will  provide  a  valuable  source  of  data  from  which  we 
may  expand  limited  general  knowledge  of  the  Logistics  Support  Analysis  process.  Since 
the  results  of  this  survey  are  to  be  used  primarily  for  academic  purposes,  a  policy  of 
nonattribution  will  apply. 

3.  All  survey  designees  are  encouraged  to  return  their  completed  survey  packages  no 
later  than  11  May  1984.  Responses  should  be  sent  to  AFALC/PTA  with  attention  to 
Mr.  Charlie  Walker.  Should  questions  arise,  please  contact  Mr.  Walker  (53754)  or 

1st  Lt  Paul  Woodland  (54437).  Your  support  is  crucial  to  the  success  of  this  data 
gathering  effort  and  is  greatly  appreciated. 

FOR  THE  COMMANDER 


1  ts  •  — > 

KENNETH  L.  MORRIS 
Acting  Director,  Logistics  Support 
Analysis 


2  Atch 

1.  Distribution  List 

2.  Survey 
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SURVEY  OF  OPINION 


The  following  series  of  questions  represent  a  survey  of  opinion  and  are  to  be  used  for 
academic  purposes  only.  This  is  an  attempt  to  collect  information  on  the  attitudes  of 
those  people  responsible  for  the  management  of  the  Logistics  Support  Analysis  process. 
Summary  reports  based  upon  individual  inputs  will  not  be  used  as  an  administrative  tool 
or  corrective  device.  A  policy  of  nonattribution  will  apply  for  the  purposes  of  this  study. 
Please  feel  free  to  answer  each  question  to  the  best  of  your  ability  and  return  your 
questionnaires  promptly.  Your  support  is  greatly  appreciated. 


SURVEY  QUESTIONS 

L  Please  list  your  specific  job  title  or  position,  (optional) 


2.  Circle  the  appropriate  rank  or  grade  level  you  presently  hold. 

a.  0-1  to  0-3  d.  GS-7  to  GS-11 

b.  0-4  to  0-5  e.  GS-12  to  GS-13 

c.  0-6  and  above  f.  GS-14  and  above 

g.  Other _ 

3.  How  long  have  you  been  assigned  in  your  current  position? 

a.  less  than  1  year  d.  3-4  years 

b.  1-2  years  e.  more  than  4  years 

c.  2-3  years 

4.  Are  you  considered  a  Logistics  Support  Analysis  (LSA)  "point -of-contact"  or  "focal 
point"  for  your  office? 

a.  yes 

b.  no 

5.  How  many  months  total  have  you  managed  or  worked  in  direct  support  of  LSA  on  this 
or  any  other  program? 


6.  Which  individual  is  also  qualified  to  respond  to  questions  about  the  LSA  process  in 
your  office? 

a.  _ (please  name) 

b.  I  am  most  qualified  to  respond 

c.  There  is  someone  in  this  organization  but  in  another  office  who  is  also  qualified 
to  answer.  (Please  name  and  provide  office  symbol) 


7.  Have  you  had  any  formal  training  and/or  education  in  the  area  of  LSA,  the  LSA  record, 
or  the  application  of  the  LSA  process? 

a.  Yes,  in  a  government  offered  or  sponsored  course.  (Please  list) 


b.  Yes,  in  a  civilian  education  program  (not  related  to  or  sponsored  by  the  government). 
(Please  list) 


c.  No,  I  have  never  had  any  formal  training  or  education  regarding  the  subject  of  LSA. 
d  Other  category  of  education.  (Please  explain) 


8.  How  effective  was  the  formal  education  or  training  in  helping  you  to  improve  management 
of  the  LSA  process  on  your  particular  program(s). 

a.  very  effective 

b.  marginally  effective 

c.  neither  effective  nor  ineffective 
d  not  effective 

9.  What  suggestions  do  you  have  in  improving  the  LSA  education  process? 


10.  How  would  you  describe  your  qualifications  to  make  decisions  or  apply  judgment  to  rate 
the  effectiveness  of  LSA  within  your  program? 

a.  fully  qualified 

b.  somewhat  (adequately)  qualified 

c.  marginally  qualified 
d  not  qualified 

11.  Which  of  the  following  job  categories  apply  to  you  (choose  the  most  applicable  answer). 

a.  I  am  an  AFSC  resource  assigned  directly  to  an  ASD  program. 

b.  I  am  an  AFLC  resource  assigned  directly  to  an  ASD  program. 

c.  More  specifically,  I  am  an  AFALC  resource  assigned  to  an  ASD  program. 

(AFALC  is  considered  a  joint-command  unit.) 

d  Other  (please  explain). 


12.  If  you  have  more  than  one  job  responsibility,  which  of  the  following  statements  is  most 
correct? 

a.  I  am  responsible  for  more  than  one  program,  but  the  responsibility  falls  primarily 
within  the  functional  area  of  LSA. 

b.  I  am  responsible  for  more  than  one  program,  but  the  responsibility  falls  primarily 
outside  the  LSA  functional  area. 

c.  I  am  responsible  for  several  functional  areas  (including  LSA),  all  under  the  same 
program. 

13.  Is  the  job  position  listed  in  Question  1  your  only  responsibility? 

a.  yes 

b.  no 

14.  Are  you  familiar  with  the  primary  guidance  documents  on  the  subject  of  LSA  program 
management?  For  example,  MIL-STD-1388-1  or  1388-1A,  AFALD's  Guide  for  Supportability 
Analysis  and  Supportability  Analysis  Record,  and  DARCOMP  750-16. 

a.  Only  MEL-STD-1388-1  or  1388-1  A. 

b.  At  least  familiar  with  two  of  the  above  documents. 

c.  I  am  not  familiar  with  the  above  documents;  however,  I  am  familiar  with  others. 
Please  list: 

d.  I  am  not  familiar  with  any  published  LSA  guidance  documents. 

15.  Which  of  the  above  listed  documents  have  been  applied  to  your  program  contract? 

Please  Identify: 

16.  Have  the  requirements  for  LSA  been  appropriately  tailored  to  meet  the  specific  needs 
of  your  program? 

a.  no 

b.  yes 

17.  Is  there  a  need  for  additional  requirements  tailoring  on  this  LSA  program? 

a.  no 

b.  yes 

18.  Do  you  feel  the  requirements  of  your  LSA  program  have  been  appropriately  levied  against 
sub-contractors  or  vendors? 

a.  no 

b.  yes 

19.  Have  you  had  any  contact  and/or  written  communication  with  the  AFALC's  Directorate 
of  Logistics  Support  Analysis  (AFALC/PTA)? 
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20.  I  spend  the  following  percentage  of  my  working  hours  on  LSA  program  related  issues. 

a.  100% 

b.  75-99% 

c.  50-74% 

d.  less  than  50% 

21.  Which  phase  of  the  "typical"  acquisition  cycle  would  you  say  your  program  most 
closely  corresponds  to? 

a.  conceptual 

b.  demonstration /validation 

c.  full-scale  development 
c L  production/deployment 

22.  Does  a  published  LSA  plan  exist  for  your  program(s)? 

a.  No 

b.  Yes,  but  it  has  not  received  government  approval. 

c.  Yes,  there  is  an  approved  LSA  plan. 

23.  Are  the  prime  contractor^)  required  by  the  statement  of  work  to  provide  you  with 
various  LSA  data  products? 

a.  No 

b.  Yes,  the  program  office  has  since  received  a  portion  of  the  required  data  products. 

c.  Yes,  but  the  program  office  has  not  received  any  of  the  required  data  products. 

24.  Is  there  a  contractual  requirement  and/or  a  plan  to  validate  and  verify  LSA  data 
submitted  to  the  government? 

a.  no 

b.  yes 

25.  Are  you  personally  satisfied  with  the  type  and  degree  of  accuracy  of  the  data  submitted 
for  program  office  review  or  reviewed  at  the  contractor's  facility. 

a.  No 

b.  Yes,  but  I  am  only  satisfied  with  the  type  of  data  being  supplied. 

c.  I  am  only  satisfied  with  the  accuracy  of  the  supplied  data. 

d.  Yes,  1  am  satisfied  with  both  the  type  and  degree  of  accuracy. 

26.  Have  government  reviews  of  LSA  data  acceptability  been  conducted  since  the  initial 
review  of  the  contractor's  proposal? 

a.  No.  Reviews  have  not  been  conducted  since  the  initial  review  of  proposals. 

b.  Neither  a  review  of  the  contractor's  proposal  or  the  LSA  data  has  been  conducted. 

c.  Yes,  at  least  one  review  of  LSA  data  acceptability  has  been  conducted. 

27.  Briefly,  how  would  you  define  the  term  "effectiveness"  as  it  applies  to  the  LSA  process? 


28.  After  close  scrutiny,  do  you  believe  the  "effectiveness"  of  your  LSA  program  could  be 
measured  in  either  quantitative  or  qualitative  terms? 


a.  no 

b.  yes 

29.  Please  identify  four  factors  which  you  feel  are  key  in  making  a  quantitative  or  qualitative 
assessment  of  LSA  process  effectiveness.  This  may  be  "overall  effectiveness"  or  "effective¬ 
ness  within  specific  functional  areas  of  the  LSA  process. 


30.  Could  the  above  factors  be  used  to  predict  the  level  of  effectiveness  for  most  LSA 
progra  ms. 

a.  no 

b.  yes 

31.  Could  LSA  effectiveness  be  appropriately  modeled  as  a  measure  of  various  predictors  or 
input  factors? 

a.  no 

b.  yes  (If  so,  how  might  the  model  be  defined  or  structured?) 


32.  How  would  you  conclude  the  following  sentence? 

In  my  office,  LSA  program  or  process  effectiveness  is  based  on  the  following  criteria: 


33.  Is  your  program  engineering  office  a  "partner"  in  the  application  and  management  of 
the  LSA  process?  If  yes,  briefly  explain  how. 

a.  no 

b.  yes 


34.  Do  other  program  functional  offices  interface  with  or  use  available  LSA  information/ 
data  products.  Examples  of  functional  offices  would  be  manufacturing,  projects,  configura¬ 
tion  control,  other  logistics  offices,  data  management,  etc. 

a.  no 

b.  yes  (How  is  this  accomplished?  Briefly  explain) 
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35.  Has  the  lack  of  involvement  by  other  functional  offices  been  detrimental  to  the  effective 
application  of  LSA? 


a.  no 

b.  yes  -  Why? 


36.  Based  on  the  four  criteria  you  cited  in  question  29,  how  would  you  rate  the  level  of 
effectiveness  for  your  particular  LSA  program(s)? 

a.  exceeding  expected  performance 

b.  meeting  expected  performance 

c.  falling  behind  expected  performance 

d.  far  behind  expected  performance 

37.  Are  there  any  specific  factors  which  have  led  to  this  level  of  LSA  effectiveness? 


38.  Can  you  identify  any  factors  that  have  detracted  from  the  LSA  program's  effectiveness? 


39.  How  might  the  above  factors  be  brought  within  the  confines  of  your  control  so  that 
improvements  to  the  LSA  process  may  be  accomplished? 


40.  Identify  any  factors  within  the  LSA  process  that  are  purely  outside  of  your  control, 
and  therefore  affect  your  ability  to  manage  the  program. 


4L  Do  you  feel  the  effectiveness  of  the  LSA  process,  as  applied  to  your  program,  could 
be  measured  by  someone  in  addition  to  yourself? 

a.  no 

b.  yes  (Please  identify  the  category  or  categories  of  individuals  who  would  be  qualified 
to  make  this  determination.) 


42.  Do  you  feel  the  input  of  individual  LSA  managers  has  or  will  contribute  to  the  overall 
knowledge  base  of  LSA  information  and  eventually  lead  toward  a  means  of  determining  LSA 
effectiveness? 

a.  no 

b.  yes 

c.  I  prefer  not  to  answer  this  question 

43.  Has  this  survey  process  provided  you  with  any  ideas  as  to  how  you  might  determine 
the  effectiveness  of  your  LSA  program  more  efficiently? 


Appendix  B:  Summary  of  Data  Responses 


SURVEY  OF  OPINION 

The  following  series  of  questions  represent  a  survey  of  opinion  and  are  to  be  used  for 
academic  purposes  only.  This  is  an  attempt  to  collect  information  on  the  attitudes  of 
those  people  responsible  for  the  management  of  the  Logistics  Support  Analysis  process. 
Summary  reports  based  upon  individual  inputs  will  not  be  used  as  an  administrative  tool 
or  corrective  device.  A  policy  of  nonattribution  will  apply  for  the  purposes  of  this  study. 
Please  feel  free  to  answer  each  question  to  the  best  of  your  ability  and  return  your 
questionnaires  promptly.  Your  support  is  greatly  appreciated. 

*  The  number  of  individual  responses  are  included  in  the  parentheses 

SURVEY  QUESTIONS 

L  Please  list  your  specific  job  title  or  position,  (optional) 


2.  Circle  the  appropriate  rank  or  grade  level  you  presently  hold. 


(4) 

a.  0-1  to  0-3 

(2) 

d. 

GS-7  to  GS-11 

(2) 

b.  0—4  to  0-5 

(16) 

e. 

GS-12  to  GS-I3 

(0) 

c.  0-6  and  above 

(0) 

f. 

GS-14  and  above 

(1)  g.  Other 


3.  How  long  have  you  been  assigned  in  your  current  position? 

(8)  a.  less  than  1  year  (2)  d.  3-4  years 

(7)  b.  1-2  years  (5)  e.  more  than  4  years 

(3)  c.  2-3  years 

4.  Are  you  considered  a  Logistics  Support  Analysis  (LSA)  "point -of-contact"  or  "focal 
point"  for  your  office? 

(22)  a.  yes 

(3)  b.  no 

5.  How  many  months  total  have  you  managed  or  worked  in  direct  support  of  LSA  on  this 
or  any  other  program? 

Less  than  12  months  (9)  Greater  than  12  months  (16) 

6.  Which  individual  is  also  qualified  to  respond  to  questions  about  the  LSA  process  in 
your  office? 

a. _ (please  name) 

J11)  b.  I  am  most  qualified  to  respond 

c.  There  is  someone  in  this  organization  but  in  another  office  who  is  also  qualified 

,  „  to,  answer.  (Please  name  and  provide  office  symbol) 

(3)  N/A  K  3 


7.  Have  you  had  any  formal  training  and/or  education  in  the  area  of  LSA,  the  LSA  record, 
or  the  application  of  the  LSA  process? 

(8)  *•  Yes,  in  a  government  offered  or  sponsored  course.  (Please  list) 


(2)  b.  Yes,  in  a  civilian  education  program  (not  related  to  or  sponsored  by  the  government). 
(Please  list) 


(13)  c.  No,  I  have  never  had  any  formal  training  or  education  regarding  the  subject  of  LSA. 
(2)  d  Other  category  of  education.  (Please  explain) 


S.  How  effective  was  the  formal  education  or  training  in  helping  you  to  improve  management 
of  the  LSA  process  on  your  particular  program(s). 

(4)  a.  very  effective 

(5)  b.  marginally  effective 

(1)  c.  neither  effective  nor  ineffective 
(1)  d  not  effective 
(14)  N/A 

9.  What  suggestions  do  you  have  in  improving  the  LSA  education  process? 


10.  How  would  you  describe  your  qualifications  to  make  decisions  or  apply  judgment  to  rate 
the  effectiveness  of  LSA  within  your  program? 

(10)  a.  fully  qualified 

(11)  b.  somewhat  (adequately)  qualified 
(3)  o  marginally  qualified 

(1)  d  not  qualified 

1L  Which  of  the  following  job  categories  apply  to  you  (choose  the  most  applicable  answer). 

(3)  a.  I  am  an  AFSC  resource  assigned  directly  to  an  ASD  program. 

(2)  b.  I  am  an  AFLC  resource  assigned  directly  to  an  ASD  program. 

(18)  c.  More  specifically,  I  am  an  AFALC  resource  assigned  to  an  ASD  program. 

(AFALC  is  considered  a  joint -command  unit.) 

(2)  d  Other  (please  explain). 


•  •  '  -  •  -  .-v  L*v 


12.  If  you  have  more  than  one  job  responsibility,  which  of  the  following  statements  is  most 
correct? 

(9)  a.  I  am  responsible  for  more  than  one  program,  but  the  responsibility  falls  primarily 
within  the  functional  area  of  LSA. 


1  A  1/  ,•* 


(2)  b.  I  am  responsible  for  more  than  one  program,  but  the  responsibility  falls  primarily 
outside  the  LSA  functional  area. 

(13)  c.  I  am  responsible  for  several  functional  areas  (including  LSA),  all  under  the  same 
program. 

(1)  N/a 

13.  Is  the  job  position  listed  in  Question  1  your  only  responsibility? 

(16)  a.  yes 
(8)  b*  no 

^r^you  familiar  with  the  primary  guidance  documents  on  the  subject  of  LSA  program 
management?  For  example,  MlL-STD-i388-I  or  1388-1A,  AFALD's  Guide  for  Supportability 
Analysis  and  Supportability  Analysis  Record,  and  DARCOMP  750-16. 

(5)  a.  Only  MIL-STD-1388-1  or  1388-1A. 

(20)  b.  At  least  familiar  with  two  of  the  above  documents. 

(0)  c.  I  am  not  familiar  with  the  above  documents;  however,  I  am  familiar  with  others. 
Please  list: 

(0)  d.  I  am  not  familiar  with  any  published  LSA  guidance  documents. 

15.  Which  of  the  above  listed  documents  have  been  applied  to  your  program  contract? 
Please  Identify: 


16.  Have  the  requirements  for  LSA  been  appropriately  tailored  to  meet  the  specific  needs 
of  your  program? 

(1)  a.  no 
(23)  b.  yes 
(1)  N/A 

17.  Is  there  a  need  for  additional  requirements  tailoring  on  this  LSA  program? 

(10)  a.  no 
(14)  b.  yes 
(1)  N/A 

18.  Do  you  feel  the  requirements  of  your  LSA  program  have  been  appropriately  levied  against 
sub-contractors  or  vendors? 

(3)  a.  no 

(21)  b.  yes 
(1)  N/A 

19.  Have  you  had  any  contact  and/or  written  communication  with  the  AFALC's  Directorate 
of  Logistics  Support  Analysis  (AFALC/PTA)? 

(1)  a.  no  65 

(22)  b.  yes 

(2)  N/A 
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20.  I  spend  the  following  percentage  of  my  working  hours  on  LSA  program  related  issues. 


(3)  a.  100% 

(5)  b.  75-99% 

(3)  c.  50-74% 

(13)d.  less  than  50% 

(1)  N/A 

2L  Which  phase  of  the  "typical"  acquisition  cycle  would  you  say  your  program  most 
closely  corresponds  to? 

(2)  a.  conceptual 

(5)  b.  demonstration/validation 
(ll)c.  full-scale  development 

(3)  <t  production/deployment 

(4)  N/A 

22.  Does  a  published  LSA  plan  exist  for  your  program(s)? 


(3)  a.  No 

W  b.  Yes,  but  it  has  not  received  government  approval. 

(16)  c.  Yes,  there  is  an  approved  LSA  plan. 

(2)  N/A 

23.  Are  the  prime  contractor^)  required  by  the  statement  of  work  to  provide  you  with 
various  LSA  data  products? 

(1)  a.  No 

(16)  b.  Yes,  the  program  office  has  since  received  a  portion  of  the  required  data  products. 
(®)  c.  Yes,  but  the  program  office  has  not  received  any  of  the  required  data  products. 


24.  Is  there  a  contractual  requirement  and/or  a  plan  to  validate  and  verify  LSA  data 
submitted  to  the  government? 


(3) 

(22) 


a.  no 

b.  yes 


25.  Are  you  personally  satisfied  with  the  type  and  degree  of  accuracy  of  the  data  submitted 
for  program  office  review  or  reviewed  at  the  contractor's  facility. 

(7)  a.  No 

(2)  b.  Yes,  but  I  am  only  satisfied  with  the  type  of  data  being  supplied. 

(1)  c.  I  am  only  satisfied  with  the  accuracy  of  the  supplied  data. 

(9)  d.  Yes,  I  am  satisfied  with  both  the  type  and  degree  of  accuracy. 

(6)  N/A 

26.  Have  government  reviews  of  LSA  data  acceptability  been  conducted  since  the  initial 
review  of  the  contractor's  proposal? 


(5)  a.  No.  Reviews  have  not  been  conducted  since  the  initial  review  of  proposals. 

(0)  b.  Neither  a  review  of  the  contractor's  proposal  or  the  LSA  data  has  been  conducted. 
(18)  c.  Yes,  at  least  one  review  of  LSA  data  acceptability  has  been  conducted. 

(2)  N/A 

27.  Briefly,  how  would  you  define  the  term  "effectiveness"  as  it  applies  to  the  LSA  process? 


28.  After  close  scrutiny,  do  you  believe  the  "effectiveness"  of  your  LSA  program  could  be 
measured  in  either  quantitative  or  qualitative  terms? 


(4)  no 

(18)  *>•  yes 

Please  identify  four  factors  which  you  feel  are  key  in  making  a  quantitative  or  qualitative 
assessment  of  LSA  process  effectiveness.  This  may  be  "overall  effectiveness"  or  "effective¬ 
ness  within  specific  functional  areas  of  the  LSA  process. 

SEE  TABLE  2 


30.  Could  the  above  factors  be  used  to  predict  the  level  of  effectiveness  for  most  LSA 
programs. 

(3)  a.  no 
(15)  b.  yes 
(7)  N/A 

3L  Could  LSA  effectiveness  be  appropriately  modeled  as  a  measure  of  various  predictors  or 
input  factors? 

(4)  a.  no 

(6)  b.  yes  (If  so,  how  might  the  model  be  defined  or  structured?) 

(15)  N/A 


32.  How  would  you  conclude  the  following  sentence? 

In  my  office,  LSA  program  or  process  effectiveness  is  based  on  the  following  criteria: 


33.  Is  your  program  engineering  office  a  "partner"  in  the  application  and  management  of 
the  LSA  process?  If  yes,  briefly  explain  how. 

(13)  a.  no 
(9)  b.  yes 
(3)  N/A 


3*.  Do  other  program  functional  offices  interface  with  or  use  available  LSA  information/ 
data  products.  Examples  of  functional  offices  would  be  manufacturing,  projects,  configura¬ 
tion  control,  other  logistics  offices,  data  management,  etc. 

a.  no 

(12)  b.  yes  (How  is  this  accomplished?  Briefly  explain) 


35.  Has  the  lack  of  involvement  by  other  functional  offices  been  detrimental  to  the  effective 
application  of  LSA? 


(10)  a.  no 
(13)  b.  yes  -  Why? 
(2)  N/A 


36.  Based  on  the  four  criteria  you  cited  in  question  29,  how  would  you  rate  the  level  of 
effectiveness  for  your  particular  LSA  program(s)? 

(1)  a.  exceeding  expected  performance 
(14)  b.  meeting  expected  performance 
(5)  c.  falling  behind  expected  performance 
(1)  d.  far  behind  expected  performance 
(4)  N/A 

37.  Are  there  any  specific  factors  which  have  led  to  this  level  of  LSA  effectiveness? 


38.  Can  you  identify  any  factors  that  have  detracted  from  the  LSA  program's  effectiveness? 


39.  How  might  the  above  factors  be  brought  within  the  confines  of  your  control  so  that 
improvements  to  the  LSA  process  may  be  accomplished? 


40.  Identify  any  factors  within  the  LSA  process  that  are  purely  outside  of  your  control, 
and  therefore  affect  your  ability  to  manage  the  program. 


41,  Do  you  feci  the  effectiveness  of  the  LSA  process,  as  applied  to  your  program,  could 
be  measured  by  someone  in  addition  to  yourself? 

(5)  a.  no 

(20)  b.  yes  (Please  identify  the  category  or  categories  of  individuals  who  would  be  qualified 
to  make  this  determination.) 


42.  Do  you  feel  the  input  of  individual  LSA  managers  has  or  will  contribute  to  the  overall 
knowledge  base  of  LSA  information  and  eventually  lead  toward  a  means  of  determining  LSA 
effectiveness? 

(3)  a.  no 
(15)  b,  yes 

(3)  c.  I  prefer  not  to  answer  this  question 

(4)  N/A 

43,  Has  this  survey  process  provided  you  with  any  ideas  as  to  how  you  might  determine 
the  effectiveness  of  your  LSA  program  more  efficiently? 

(17)  a.  no 
(8)  b.  yes 
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sub-contractors  or  vendors? 
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(21)  b.  yes 
(1)  N/A 
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of  Logistics  Support  Analysis  (AFALC/PTA)? 

(1)  a.  no  65 

(22)  b.  yes 

(2)  N/A 
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